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EXECUTIVE SUMMARY 

This study examined the impact of NISL’s Executive Development Program for 
principals on student achievement in Pennsylvania schools between 2006-2009. Roughly half of 
the NISL-trained principals started the program during the 2007 school year and completed it in 
the 2008 school year, whereas the other half started during the 2008 school year and completed it 
in the 2009 school year. Schools served by principals participating in the Executive 
Development Program were individually matched to comparison schools with similar school 
performance and demographic profiles in 2006, which served as the baseline year. For 
elementary schools, it was possible to make individual school matches within the same school 
district for 36 school pairs. An additional 32 elementary schools were included in an out-of- 
district matched comparison sample. For all middle and high schools, it was necessary to match 
outside the school district. The percentages of students achieving proficient or above in 
mathematics and reading or English/Language Arts (ELA) across all grade levels were used to 
create aggregate school performance indices for each year 2006-2009. Repeated-measures 
analyses were performed to determine whether there were differences in school performance 
trends between schools served by NISL-trained principals and matched comparison schools. 

At the elementary level, statistically significant differences in school performance trends 
were observed between the matched within-district NISL and comparison schools in both 
mathematics and reading/ELA. For both subjects, NISL schools had statistically significantly 
higher rates of improvement in school performance than did comparison schools. Further, a 
significant quadratic effect was observed for mathematics, indicating an acceleration in the rate 
of improvement for NISL schools. In terms of unadjusted results, NISL schools gained 3.9% 
versus 0.7% for comparison sites in mathematics, and 4.1% versus 3.7% in reading/ELA. For 
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the matched out-of-district elementary schools, significant trends favoring NISL over 
comparison schools were also indicated in both mathematics and reading/ELA. Significant 
quadratic effects in mathematics further revealed acceleration in the rate of growth over time for 
NISL schools. 

As with the elementary school results, statistically significant positive effects of NISL 
status were observed for both mathematics and reading in the middle school sample. Unadjusted 
results showed that NISL students improved the percentage of students achieving proficiency in 
mathematics by 5% from 2006 to 2009, compared to a 1% improvement in comparison schools. 
In reading/ELA, NISL middle schools improved the percentage of students achieving proficiency 
by 3% versus 1% in comparison schools. As with elementary schools, a significant quadratic 
effect on mathematics school performance trends revealed that the rate of improvement was 
accelerating over time in NISL schools. 

NISL high schools had statistically significantly higher rates of improvement, with 
adjusted differences in 2009 performance equal to 5%. In math, on an unadjusted basis, NISL 
high schools improved the percentage of students reaching proficiency by 5.9% from 2006 to 
2009, compared to a decline of 0.5% in comparison schools. In reading/ELA, performance in 
NISL sites remained relatively constant (unadjusted difference of +0.6% between 2006 and 
2009), while performance in comparison sites declined by 3.1% over the same time period. 
Overall conclusions are that across the three school levels and two subjects, NISL schools 
consistently surpassed the comparison schools at a statistically significant level in achievement 
gains from the baseline year of 2006 to 2009. Predictably, the gains were strongest in 2008 and 
2009 as levels of participation in the NSL program (both number of principals and exposure) 
increased. The significant quadratic effects reflected this trend for program impacts to accelerate 
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over time. Across the four school cohorts examined (matched within-district elementary schools, 
matched out-of-district elementary schools, middle schools, and high schools), NISL schools 
surpassed comparison schools in percentage achieving proficiency in mathematics by 2.69%, 
3.71%, 1.70%, and 5.52%, respectively, and in reading/ELA by .37%, 2.55%, 1.63%, and 1.89%, 
respectively. Given that approximately 40,000 students were included in the combined samples, 
these advantages appear highly educationally meaningful. For example, across the high school 
subsample alone, replication of the NSL effects in mathematics and reading/ELA would result in 
about 275 and 103 more students achieving proficiency on the respective tests. Given the limited 
amount of post-program program participation for the majority of principals prior to the 2008 
and 2009 achievement assessments, follow-up evaluation research is encouraged to determine 
program effects over a longer time period. 
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INTRODUCTION 

The National Institute of School Leadership’s (NISL's) Executive Development Program was 
established to train school leaders to drive their schools to high performance. The program 
emphasizes the role of principals as strategic thinkers, instructional leaders, and creators of a 
just, fair, and caring culture in which all students meet high standards. Its primary goal is to 
ensure that the participating school leaders have the knowledge, skills, and tools to effectively set 
direction for teachers, support their staffs, and design an efficient organization. The curriculum, 
which was designed by experts on leadership training across a number of fields, was developed 
with an $1 1 million investment and five years of research and piloting. Professional development 
goals are to provide high-quality instruction (both online and face-to-face), an advanced 
research-based curriculum, and an interactive approach to learning that includes simulations, 
case studies, school evaluations, and online activities. 

Key expectancies for NISL-trained principals include: 

• Formulating a clear vision to inspire others in the school communities, 

• Implementing fully-aligned, standards-based instructional systems, 

• Building effective instructional programs in the core academic subjects, particularly 
math, language arts and science, 

• Using data to produce continuous improvements in instruction and student achievement, 

• Providing effective training programs to build a professional learning community for 
school faculty and staff, and 

• Creating integrated school improvement plans that reflect strategic and systemic thinking. 
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The curriculum is organized into four courses: World-Class Schooling (Principal as a Strategic 
Thinker and School Designer, Standards-Based Instruction), Teaching and Learning, Developing 
Capacity and Commitment, and Driving for Results. Training sessions are designed to be highly 
interactive through the use of simulations and assignment of “pre-work” and “homework” to 
participants. 

Prior evaluations of the Executive Development Program have shown that the NISL 
program can be economically implemented with high fidelity (Meristem Group, 2009). 
Importantly, positive student achievement patterns have been associated with program 
participation by school leaders. However, these prior studies have used descriptive or 
correlational designs lacking comparison groups or strong controls over sample selection bias. 
Accordingly, to provide more rigorous evidence and support causal conclusions regarding 
program impacts, the present longitudinal study of student achievement in Pennsylvania schools, 
from 2006-2009, was conducted . A carefully matched comparison-group ex post facto design 
was employed in which schools served by principals participating in the program were 
individually matched to control schools with similar school performance and demographic 
profiles in the baseline (pre-program) year of 2006. The specific research questions addressed 
were: 

1. How do the trends in school level performance in mathematics differ between schools 
served by NISL-trained principals and matched comparison schools at the elementary, 
middle, and high school levels? 

2. How do the trends in school level performance in reading and English/Language Arts 
(ELA) differ between schools served by NISL-trained principals and matched 
comparison schools at the elementary, middle, and high school levels? 
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METHOD 

Sample 

Data from all Pennsylvania elementary schools with complete test score data from 2005- 
2006 through 2008-2009 were initially considered for inclusion in the analyses. There were a 
total of 70 NISL elementary schools, 19 NISL middle schools, and 12 NISL high schools. As 
explained below, 36 of the NISL elementary schools were included in a within-district matched 
samples analysis, and 32 were included in a separate set of analyses based on an out-of-district 
matched comparison sample. In the elementary school sample, 19 of the NISL principals 
completed the NISL program in 2009, and 17 completed it in 2008. In the middle school 
sample, 7 of the NISL principals completed the NISL program in 2009, and 12 completed it in 
2008. In the high school sample, 6 of the NISL principals completed the NISL program in 2009, 
and 8 completed it in 2008. Thus, roughly half of NISL principals at any given grade level 
started the program in 2007 and completed in 2008, while half started in 2008 and 2009. The 
elementary within-district matched sample included an average of 4,565 students in comparison 
schools each year, and 5,898 students in NISL schools. Corresponding comparison and NISL 
average annual student sample sizes were 5,233 and 4,847 for elementary out-of-district matched 
samples, 8,916 and 7,498 for middle school out-of-district matched samples, and 3,017 and 
2,552 for high school out-of-district matched samples. 

Participation by principals occurred through a multi-step process. Initially, the State 
Education Agency (SEA) publicized NISL to district superintendents. The latter, in turn, 
identified principals and assistant principals in their districts and encouraged them to apply. The 
potential applicants were characterized by the SEA and superintendents as mixed in leadership 
potential (experiences, accomplishments, and skills), with some demonstrating strong promise 
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and others regarded as needing professional development support to improve instructional 
leadership skills. Actual applicants were selected by regional coordinators using an evaluation 
rubric (see Appendix B). During the first two years of the program, participation was limited to 
principals or assistant principals in their first three years on the job. The highest weighting on 
the rubric evaluation was given to candidates from the lowest performing schools. Starting on 
January 1, 2008, a new state (Act 45 of 2007) policy requiring a principals’ induction program 
went into effect. Based on the law’s requirement that all school and system leaders meet 
approved continuing education requirements, all applicants henceforth were accepted to the 
program on a first-come-first- served basis. This change in recruitment practices, however, did 
not affect the present sample of NISL participants. 

Elementary school matching procedure. A principal components analysis using 2006 
school performance index values in mathematics and reading (see below) and the 2006 
percentage of students classified as economically disadvantaged was performed to construct a 
regression-based factor score to use to identify matched pairs of schools for the analyses. Each 
NISL school was individually matched to a comparison school in the same school district that 
had the closest factor score. Matches were considered suitable only if the factor scores were 
within +/-0.25 standard deviation units. The matching process yielded a final sample for analysis 
of 36 NISL and 36 comparison schools at the elementary level that had matches within the same 
school district. Of the remaining 34 NISL elementary schools, two did not have test score data at 
all four time points. The remaining 32 were matched out-of-district, and these analyses were 
conducted separately. As shown in Table 1, this matching process yielded very closely matched 
samples. In 2006, the NISL elementary schools with within-district matches had a slightly 
lower percentage of economically disadvantaged students (15.8% versus 17.8%), slightly higher 
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percentages of students with limited English proficiency (18.4% versus 14.8%), and a lower 
percentage of students who were proficient in math (78.0% versus 80.5%). For the out-of- 
district matches, all matching variables were within 0.1%, except percentage of limited English 
proficient students served (1.1% in comparison sites versus 0.6% in NISL sites). Table Al.A in 
the appendix provides 2006 characteristics for each pair in the within-district matched samples, 
and Table Al.B provides this information for the out-of-district matched pairs. 

Secondary school matching procedure. It was not possible to individually match middle 
and high school NISL schools to a comparison school within the same school district — in many 
cases, the NISL secondary school was the only school at that level within the district. At the 
middle and high school levels, an out-of-district match was made to each NISL school by 
matching the NISL school to a comparison school with the closest factor score. There were 19 
NISL middle schools and 14 NISL high schools. As shown in Table 1, the matching process led 
to relatively well-matched samples for the middle school analyses, although the NISL sample 
was slightly more economically disadvantaged (27.6% versus 24.6%), and had somewhat lower 
initial scores in math (68.2% versus 72.0% proficient) and reading (69.9% versus 73.2%). Table 
A2 presents characteristics for each middle school pair in 2006. Likewise, the high school 
samples were relatively well-matched, with comparison schools having somewhat higher 
percentages of economically disadvantaged students (M = 27.9% versus 23.9%), higher 
proficiency rates in mathematics (M = 47.8% versus 45.0%), and higher proficiency rates in 
Reading/ELA (M = 63.8% versus 59.6%). NISL schools served higher percentages of special 
education students (14.35 versus 10.9%). Table A3 summarizes characteristics for each high 



school pair in 2006. 
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Measures 

School demographics. The proportions of students in tested grade levels (3-8 and 11) 
who were economically disadvantaged, received special education services as evidenced by the 
existence of an individualized education plan (IEP), or who were classified as having limited 
English proficiency (LEP) were computed for each school. In addition to be used to establish 

Table 1 

Selected 2006 Characteristics of NISL and Comparison Schools 



School Type 


Economically 
Disadvantaged % 


IEP % 


LEP % 


Math % 
Proficient 


Reading/ELA 
% Proficient 


Elementary 1 


Comparison 


17.81 


6.41 


14.8 


80.49 


69.20 


NISL 


15.82 


6.61 


18.4 


78.04 


68.10 


Elementary 2 


Comparison 


17.0 


7.6 


1.1 


79.0 


68.7 


NISL 


16.9 


7.5 


0.6 


78.9 


68.5 


Middle 2 


Comparison 


24.6 


12.1 


0.5 


72.0 


73.2 


NISL 


27.6 


13.5 


0.5 


68.2 


69.9 


High 2 


Comparison 


27.9 


10.9 


0.2 


47.8 


63.8 


NISL 


23.9 


14.3 


0.2 


45.0 


59.6 



~~ I ^ 

Within-district matches. “Out-of-district matches. N - 36 NISL and comparison schools for elementary 
with within-district matches, N = 32 of each type for elementary out-of-district matches, N = 19 of each 
for middle, and N = 14 of each for high. 
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matches, these variables were incorporated as covariates in the inferential statistical models. 

School performance indices. Summary indices of school performance were constructed 
for both reading/ELA and mathematics by computing the proportion of students at all tested 
grade levels (grades 3 to 8 and grade 1 1) who scored proficient or higher on the Pennsylvania 
State Assessment. Note that for high schools, test scores were available only for eleventh grade. 
Analyses 

At each grade-level configuration (elementary, middle, and high school) and for each 
subject area (reading and math), a repeated-measures analysis were conducted with 2006, 2007, 
2008, and 2009 school performance index values as the dependent variable. NISL status was 
treated as the independent variable. Covariates included the 2006 proportions of students who 
were economically disadvantaged, had IEPs, or who were designated as limited English 
proficient. School performance index values in 2006 were also included as covariates to 
facilitate comparison of trend lines. The analyses were weighted based on the average number of 
students tested per year for each school between 2006 and 2009 in each respective subject. 
Importantly, the repeated-measures analyses employed and the fact that there was low principal 
mobility (only 4 NISL principals, or about 4%, were re-assigned to different schools at any point 
from 2006-2009) during the period of the study provided direct control over sampling bias, 
which is often a strong validity threat in evaluating school and teacher leadership programs. 

That is, the achievement trajectories analyzed for NISL and non-NISL schools reflected pre- and 
post-program outcomes almost exclusively associated with the same school leaders. Thus, 
essentially, each school leader served as his/her own control for analyzing longitudinal 



achievement patterns. 




School Performance Trends in Pennsylvania 13 



Mauchly’s test of sphericity was employed to evaluate the assumption of sphericity, and 
Levene’s test was performed to test the assumption of equal variances for the 18 sets of analysis 
(3 school levels x 3 years x 2 subjects). Box’s test of equality of covariance matrices was 
performed. The criterion for statistical significance of the diagnostic tests was set at a = .01. 

The criterion for statistical significance of the inferential test of the program X testing occasion 
effect (i.e., the test of the effect of NISL status on school performance trend lines) was set at a = 
.05. 

In all but 1 of the 24 diagnostic analyses, the results of the Levene, Mauchly or Box tests 
were all significant (p’s < .01), indicating a violation of assumptions. Accordingly, the 
Greenhouse-Geiser corrected degrees of freedom was employed in tests of statistical 
significance. The Greenhouse-Geiser correction provides for a conservative, robust test of 
significance in repeated-measures models in which the foregoing assumptions appear to be 
violated. A summary of the Levene test results is provided in Table 2. 

Summary effects. Summary effects were computed for each set of analyses as the 
difference in covariate-adjusted 2009 school performance index values in mathematics and 
reading/ELA. Thus, the summary effects measures indicate the gain or decline in the percentage 
of students achieving proficient or higher attributable to NISL participation. 
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Table 2 

Results of Levene’s tests of equal variances 



Subject and Year of Testing 


School Level 
and Results 


Mathematics 




Reading 


Elementary 
(w/in district) 


2007 


2008 


2009 




2007 


2008 


2009 


df 


1, 10467 


1, 

10467 


1, 10467 




1, 10455 


1, 10455 


1, 10455 


F 


228.5 


239.6 


114.5 




58.8 


104.7 


3.39 


Prob. 


<.001 


<.001 


<.001 




<.001 


<.001 


.06 


Elementary 
(out of 
district) 


2007 


2008 


2009 




2007 


2008 


2009 


df 


1, 10088 


1, 

10088 


1, 10088 




1, 10077 


1, 10077 


1, 10077 


F 


129.6 


0.5 


0.4 




344.7 


724.6 


655.8 


Prob. 


<.001 


.50 


.55 




<.001 


<.001 


<.001 


Middle 


2007 


2008 


2009 




2007 


2008 


2009 


df 


1, 16128 


1, 

16128 


1, 16128 




1, 16416 


1, 16416 


1, 16416 


F 


916.4 


771.7 


37.7 




654.9 


728.2 


1727.5 


Prob. 


<.001 


<.001 


<.001 




<.001 


<.001 


<.001 


High 


2007 


2008 


2009 




2007 


2008 


2009 


df 


1, 5796 


1, 5796 


1, 5796 




1, 5795 


1, 5795 


1, 5795 


F 


356.3 


20.4 


172.7 




315.1 


564.5 


50.7 


Prob. 


<.001 


<.001 


<.001 




<.001 


<.001 


<.001 



Note. Box and Mauchley test results were significant, indicating violations of assumptions in all 
analyses. 



RESULTS 

Elementary Schools: Within-district Matched Samples 



Mathematics 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
80.8 for comparison schools and 78.8 for NISL schools. As shown in Table 3, by 2009 NISL 




School Performance Trends in Pennsylvania 15 



schools (M =81.7) had surpassed comparison schools (M = 81.5). Figure 1 displays the 
observed trend lines in math school performance index values for each group. As depicted, the 
trend lines cross, indicating greater gains over time for NISL schools. 

Table 3 

Mean Percentage of Students Scoring Proficient or Higher in Mathematics by School Type and 
Year, Elementary Schools (Within-district Matched Samples) 



School 










Type 


2006 


2007 


2008 


2009 


NISL 


78.8 


80.8 


80.3 


81.7 


Comparison 


80.8 


82.0 


81.1 


81.5 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 

Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 
in math school performance index values over time (F 2 . 7, 23659 = 620.2, p < . 01 ), suggesting that 
the percentage of students achieving proficiency in mathematics increased over time across 
schools in the analysis sample. The economically-disadvantaged (F 2 . 7, 23659 = 289.8, p < .01), 
LEP (F 2 7,23659 = 225.4, p < .01), and IEP (F 2 . 7, 23659 = 94 . 1,77 < .01) covariates were also 
statistically significant predictors of trend. A statistically significant NISL status X math within- 
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Figure 1. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in Mathematics 
by School Type (NISL, comparison) and Year, Elementary Schools (Within-district Matched 
Samples) 

subjects interaction effect was observed (F 2 . 7, 23659 = 1 1 9.3, p < .001), indicating that the trend 
lines in school-level math performance were not equal in NISL and comparison sites. In addition 
to the significant difference in the linear component of the trend lines, within-subjects contrasts 
indicated a statistically significant quadratic effect of NISL status on math score trends (L 1,10463 = 
68.2, p c.001). As shown in Ligure 2, the significant linear effect indicates that, across the time 
period, NISL schools gained at a greater average rate than comparison schools. The statistically 
significant quadratic effect indicates an acceleration in the rate of gain over time for NISL 



schools. 
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School Type 

Comparison 

— NISL 



Figure 2. Covariate Adjusted Trend Lines in Math School Performance Index Values by School 
Type, 2006-2009 Elementary Schools (Within-district Matched Samples). 

Reading 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
70.0 for comparison schools and 69.5 for NISL schools. As shown in Table 4, by 2009 NISL 
schools ( M = 73.6) had virtually equal reading performance as comparison schools ( M = 73.7). 
Ligure 3 displays the observed trend lines in reading/ELA school performance index values for 



each group. As shown, the trend lines were nearly parallel, although comparison schools had a 
greater “dip” in performance in 2008 than did NISL schools. 
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Table 4 

Mean Percentage of Students Scoring Proficient or Higher in Reading by School Type and Year, 
Elementary Schools 



School 










Type 


2006 


2007 


2008 


2009 


NISL 


69.5 


73.9 


72.3 


73.6 


Comparison 


70.0 


73.8 


71.4 


73.7 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 




Figure 3. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in Reading by 
School Type (NISL, comparison) and Year, Elementary Schools (Within-district Matched 
Samples). 



Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 
in reading/ELA school performance index values over time ( F 2 a . 25135 = 480.6, p < .001), 
suggesting that the percentage of students achieving proficiency in reading/ELA increased over 
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time across schools in the analysis sample. The economically-disadvantaged (F 2 a, 25135 = 226.0, 
p < .001),), LEP (F 2 . 4, 25135 = 261.9, p < .01), and IEP (^ 2 . 4, 25135 = 65.9, p < .01) covariates were 
also statistically significant predictors of trend. A statistically significant NISL status X 
reading/ELA within-subjects interaction effect was observed (F 2 a, 25135 = 28.7, p < .01), 
indicating that the trend lines in school-level reading/ELA performance were not equal in NISL 
and comparison sites. In addition to the significant difference in the linear component of the 
trend lines, within-subjects contrasts indicated a statistically significant cubic effect of NISL 
status on math score trends (Fi , 10451 = 42.5, p c.001). As shown in Figure 4, the significant linear 
effect indicates that, across the time period, NISL schools gained at a greater average rate than 
comparison schools after controlling for school demographics. The statistically significant cubic 
effect was produced because NISL schools had a statistically significantly lower rate of decline 
from 2007 to 2008 relative to comparison schools. 
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School Type 

Comparison 

— NISL 



Figure 4. Covariate-adjusted Trend Lines in Reading/ELA School Performance Index Values by 
School Type, 2006-2009 Elementary Schools (Within-district Matched Samples). 

Elementary Schools: Out-of-district Matched Samples 

Mathematics . 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
78.6 for comparison schools and 78.5 for NISL schools. As shown in Table 5, by 2009 NISL 
schools (M =81.4) had surpassed comparison schools (M = 77.9). Figure 5 displays the 
observed trend lines in math school performance index values for each group. 
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Table 5 

Mean Percentage of Students Scoring Proficient or Higher in Mathematics by School Type and 
Year, Elementary Schools (Out-of-district Matched Samples). 



School 










Type 


2006 


2007 


2008 


2009 


NISL 


78.5 


80.6 


79.6 


81.4 


Comparison 


78.6 


77.0 


77.6 


77.9 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 




Figure 5. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in Mathematics 
by School Type (NISL, comparison) and Year, Elementary Schools (Out-of-district Matched 
Samples). 

Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 
in math school performance index values over time (F 2 . 8,28499 = 125 9.6 ,p < .001), suggesting 
that the percentage of students achieving proficiency in mathematics increased over time across 
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schools. The economically-disadvantaged (F 2 . 8 , 28499 = 74.2, p < .001), LEP (F 2 .%, 28499 = 260.3, p 
< .001), and IEP (F 2 . 8, 28499 = 27.1, p < .001) covariates were also statistically significant 
predictors of trend. A statistically significant NISL status X math within-subjects interaction 
effect was observed (Fo.s , 28499 = 596.1, p < .001), indicating that the trend lines in school-level 
math performance were not equal in NISL and comparison sites. In addition to the significant 
difference in the linear component of the trend lines, within-subjects contrasts indicated a 
statistically significant quadratic effect of NISL status on math score trends (Fuoo 84 = 203.8, p 
c.001). As shown in Figure 6, the significant linear effect indicates that, across the time period, 
NISL schools gained at a greater average rate than comparison schools. The statistically 
significant quadratic effect indicates an acceleration in the rate of gain over time for NISL 
schools. 
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School Type 

Comparison 

NISL 



Figure 6. Covariate Adjusted Trend Lines in Math School Performance Index Values by School 
Type, 2006-2009 Elementary Schools (Out-of-district Matched Samples). 

Reading 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
69.0 for comparison schools and 68.1 for NISL schools. As shown in Table 6, by 2009 NISL 
schools ( M = 72.0) had higher reading performance than comparison schools (M = 70.4). Ligure 
7 displays the observed trend lines in reading/ELA school performance index values for each 
group. As shown, comparison school performance remained virtually unchanged during the 
period 2006-2009, whereas performance improved substantially in NISL schools. 
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Table 6 

Mean Percentage of Students Scoring Proficient or Higher in Reading by School Type and Year, 
Elementary Schools (Out-of-district Matched Samples). 



School 










Type 


2006 


2007 


2008 


2009 


NISL 


68.1 


71.8 


70.8 


72.0 


Comparison 


70.0 


69.9 


70.1 


70.4 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 




Figure 7. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in Reading by 
School Type (NISL, comparison) and Year, Elementary Schools (Out-of-district Matched 
Samples). 

Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 
in reading/ELA school performance index values over time (Fi.y,. 27774 = 283.7, p < .001), 
suggesting that the percentage of students achieving proficiency in reading/ELA increased over 
time across schools in the analysis sample. The economically-disadvantaged (F 2 . 8, 27774 = 25.4, p 
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< .001), LEP (F 2 . 8 , 27774 = 204.3, p < .001), and IEP (F 2 . 8 , 27774 = 220.0, p < .001) covariates were 
also statistically significant predictors of trend. A statistically significant NISL status X 
reading/ELA within-subjects interaction effect was observed (F 2 . 8 , 27774 = 274.9, p < .001), 
indicating that the trend lines in school-level reading/ELA performance were not equal in NISL 
and comparison sites. In addition to the significant difference in the linear component of the 
trend lines, within-subjects contrasts indicated a statistically significant cubic effect of NISL 
status on reading/ELA score trends (Lj. 10073 = 153.1, p c.001). As shown in Ligure 8, the 
significant linear effect indicates that, across the time period, NISL schools gained at a greater 
average rate than comparison schools after controlling for school demographics. The statistically 
significant cubic effect was produced because NISL schools had a slight dip in performance from 
2007 to 2008 relative to comparison schools. 




School Type 

Comparison 

NISL 



Ligure 8. Covariate-adjusted Trend Lines in Reading/ELA School Performance Index Values by 
School Type, 2006-2009 Elementary Schools (Out-of-district Matched Samples). 
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Middle Schools 

Mathematics 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
67.5 for comparison schools and 71.5 for NISL schools. As shown in Table 7, by 2009 NISL 
schools (M = 76.4) had increased proficiency rates by about 5%, whereas comparison schools 
had improved by about 1% ( M = 68.5). Figure 9 displays the observed trend lines in math school 
performance index values for each group. 

Table 7 

Mean Percentage of Students Scoring Proficient or Higher in Mathematics by School Type and 
Year, Middle Schools 



Column 1 


2006 


2007 


2008 


2009 


NISL 


71.5 


73.1 


73.4 


76.4 


Comparison 


67.5 


67.1 


68.5 


68.5 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 
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Figure 9. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in Mathematics 
by School Type (NISL, comparison) and Year, Middle Schools 

Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 
in math school performance index values over time (F 1 . 9 , 32033 = 1655.3, p < .01), suggesting that 
the percentage of students achieving proficiency in mathematics increased. The economically- 
disadvantaged (Fi. 9> 32033 = 1293.8, p < .01), LEP (Fi. 9> 32033 = 1102.7, p < .01), and IEP (F L9 , 32033 
= 95.2, p < .01) covariates were also statistically significant predictors of trend. A statistically 
significant NISL status X math within-subjects interaction effect was observed (Fi. 9,32033 = 320.9, 
p < .01), indicating that the trend lines in school-level math performance were not equal in NISL 
and comparison sites. In addition to the significant difference in the linear component of the 
trend lines, within-subjects contrasts indicated a statistically significant quadratic effect of NISL 
status on math score trends (Fi, 16422 = 19.64, p < .001). As shown in Figure 10, the significant 
linear effect indicates that, across the time period, NISL schools gained at a greater average rate 
than comparison schools. The statistically significant quadratic effect indicates an acceleration 
in the rate of gain over time for NISL schools. 
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School Type 

Comparison 

NISL 



Figure 10. Covariate Adjusted Trend Lines in Math School Performance Index Values by 
School Type, 2006-2009 Middle Schools. 

Reading/ELA 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
69.5 for comparison schools and 72.2 for NISL schools. As shown in Table 8, by 2009 NISL 
schools ( M = 75.2) had increased proficiency rates by about 3%, whereas comparison schools 
had improved by about 1% ( M = 70.7). Figure 1 1 displays the observed trend lines in 
reading/ELA school performance index values for each group. 
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Table 8 

Mean Percentage of Students Scoring Proficient or Higher in Reading/ELA by School Type and 
Year, Middle Schools 



School 










Type 


2006 


2007 


2008 


2009 


NISL 


72.2 


71.9 


73.8 


75.2 


Comparison 


69.7 


69.9 


69.0 


70.7 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 




Figure 11. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in Reading by 
School Type (NISL, comparison) and Year, Middle Schools 

Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 
in reading/ELA school performance index values over time (F 2 . 9, 47970 = 1759.4, p < .001), 
suggesting that the percentage of students achieving proficiency in reading/ELA increased. The 
economically-disadvantaged (F 2 . 9, 47970 = 1010.9, p < .001), LEP (F 2 . 9, 47970 = 422.3, p<. 001), 
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and IEP (F2.9, 47970 = 687.0, p < .01) covariates were also statistically significant predictors of 
trend. A statistically significant NISL status X reading/ELA within-subjects interaction effect 
was observed (F2.9, 47970 = 1196.2 ,p < .001), indicating that the trend lines in school-level 
reading/ELA performance were not equal in NISL and comparison sites. In addition to the 
significant difference in the linear component of the trend lines, within-subjects contrasts 
indicated a statistically significant quadratic effect of NISL status on reading/ELA score trends 
(Li , 16412 = 43.05, p < .001). As shown in Ligure 12, the significant linear effect indicates that, 
across the time period, NISL schools gained at a greater average rate than comparison schools. 
The statistically significant quadratic effect indicates an acceleration in the rate of gain over time 
for NISL schools. 
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School Type 

Comparison 

— NISL 



Figure 12. Covariate-adjusted Trend Lines in Reading/ELA School Performance Index Values 
by School Type, 2006-2009 Middle Schools. 

High Schools 



Mathematics 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
44.9 for comparison schools and 46.2 for NISL schools. As shown in Table 9, by 2009 NISL 
schools (M = 52.1) had increased proficiency rates by about 5%, whereas comparison schools 
declined by about 0.5% ( M = 44.4). Ligure 13 displays the observed trend lines in math school 
performance index values for each group. 
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Table 9 

Mean Percentage of Students Scoring Proficient or Higher in Mathematics by School Type and 
Year, High Schools 



School 










Type 


2006 


2007 


2008 


2009 


NISL 


46.2 


49.4 


53.6 


52.1 


Comparison 


44.9 


43.2 


46.3 


44.4 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 




Figure 13. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in 
Mathematics by School Type (NISL, comparison) and Year, High Schools 

Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 

in math school performance index values over time {F^a, 14139 = 624.8, p < .001), suggesting that 

the percentage of students achieving proficiency in mathematics increased. The economically- 



disadvantaged (F2.4, 14139 = 37.7, p < .001), LEP (F2.4. 14139 = 158.9, p < .001), and IEP (F 2 a. 14139 
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= 168.4, p < .001) covariates were also statistically significant predictors of trend. A statistically 
significant NISL status X math within-subjects interaction effect was observed (Fia. 14139 = 548.4, 
p < .001), indicating that the trend lines in school-level math performance were not equal in 
NISL and comparison sites. In addition to the significant difference in the linear component of 
the trend lines, within-subjects contrasts indicated a statistically significant quadratic effect of 
NISL status on math score trends (^5792 = 1316.6, p < .001). As shown in Figure 14, the 
significant linear effect indicates that, across the time period, NISL schools gained at a greater 
average rate than comparison schools. The statistically significant quadratic effect indicates a 
deceleration in the rate of gain over time for NISL schools. 




School Type 

Comparison 

— NISL 



Figure 14. Covariate Adjusted Trend Lines in Math School Performance Index Values by 
School Type, 2006-2009 High Schools. 
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Reading 

Descriptive results. The unadjusted mean school performance index values in 2006 were 
64.9 for comparison schools and 59.6 for NISL schools. As shown in Table 10, by 2009 NISL 
schools ( M = 60.2) had increased proficiency rates by 0.6%, whereas comparison schools 
declined by about 3% ( M = 61.8). Figure 15 displays the observed trend lines in reading/ELA 
school performance index values for each group. 

Table 10 

Mean Percentage of Students Scoring Proficient or Higher in Reading/ELA by School Type and 
Year, High Schools 



School 










Type 


2006 


2007 


2008 


2009 


NISL 


59.6 


62.0 


61.2 


60.2 


Comparison 


64.9 


60.0 


63.6 


61.8 



Note. Weighted by number of students tested, so figures may not correspond to unweighted 
means presented in Table 1. 




Figure 15. Unadjusted Mean Percentage of Students Scoring Proficient or Higher in 
Reading/ELA by School Type (NISL, comparison) and Year, High Schools 
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Inferential tests. Tests of within-subjects effects revealed a statistically significant trend 
in reading/ELA school performance index values over time {Fja. 14121 = 403.3, p < .001), 
suggesting that the percentage of students achieving proficiency in reading/ELA declined over 
time across schools. The economically-disadvantaged (F2.4, 14121 = 40.1 ,p < .001), LEP (F2.4, 

14121 = 354.2, p < .001), and IEP (F2.4, 14121 = 32.5, p < .001) covariates were also statistically 
significant predictors of trend. A statistically significant NISL status X reading/ELA within- 
subjects interaction effect was observed (F2.4. 14139 = 344.1, p < .001), indicating that the trend 
lines in school-level reading/ELA performance were not equal in NISL and comparison sites. In 
addition to the significant difference in the linear component of the trend lines, within-subjects 
contrasts indicated a statistically significant quadratic effect (/'Y5791 = 944.9, p < .001) and cubic 
effect (/^i ,5791 = 321.2, p < .001) of NISL status on reading/ELA score trends. As shown in 
Figure 16, the significant linear effect indicates that, across the time period, NISL schools 
declined at a slower overall rate than comparison schools. The quadratic effect indicates a 
greater average acceleration in decline for NISL schools from 2007-2009, whereas the cubic 
effect is reflected in the zig-zag pattern of comparison school results. 
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School Type 

Comparison 

— NISL 



Figure 16. Covariate-adjusted Trend Lines in Reading/ELA School Performance Index Values 
by School Type, 2006-2009 High Schools. 
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FINDINGS AND DISCUSSION 

Study Overi’iew 

This study examined the impact of NISL’s Executive Development Program for 
principals on student achievement in Pennsylvania schools between 2006-2009. The 2006 
school year was treated as the baseline year for the analysis. Roughly half of the NISL-trained 
principals started the program during the 2007 school year and finished in the 2008 school year, 
whereas the other half started during the 2008 school year and finished in the 2009 school year. 
Schools served by principals participating in the Executive Development Program were 
individually matched to comparison schools with similar school performance and demographic 
profiles in 2006. For elementary schools, it was possible to make individual school matches 
within the same school district for 36 school pairs. An additional 32 elementary schools were 
included in an out-of-district matched comparison sample. For all middle and high schools, it 
was necessary to match outside the school district. The percentages of students achieving 
proficient or above in mathematics and reading or English/Language Arts (ELA) across all grade 
levels were used to create aggregate school performance indices for each year 2006-2009. 
Repeated-measures analyses were performed to determine whether there were differences in 
school performance trends between schools served by NISL-trained principals and matched 
comparison schools. 

Findings 

Summary effects. As shown in Figure 17, NISL schools had higher-than-expected 
performance in 2009 relative to comparison schools at all grade levels in both subject areas. The 
largest differences between the percentages of NISL and comparison students achieving 
proficiency were in mathematics: +2.69%, +3.71%, +1.70%, and +5.52% for elementary within- 
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district, elementary out-of-district, middle schools, and high schools, respectively. Smaller, but 
statistically significant gains were observed for reading/ELA: +0.37%, +2.55%, +1.63%, and 
+1.89%, the four school cohorts, respectively. 




Figure 17. NISL Effects: 2009 Covariate-adjusted Differences in Percentages of Students 
Achieving Proficiency Relative to Comparison Schools . 

Elementary Schools. For the matched within-district elementary school sample, 
statistically significant differences in school performance trends were observed between NISL 
and comparison schools in both mathematics and reading/ELA. In both cases, NISL schools had 
statistically significantly higher rates of improvement in school performance than did 
comparison schools. After controlling for differences in school demographics and 2006 school 
performance, NISL elementary schools had about 2% more students achieve proficiency in math 
than did comparison schools in 2009. The adjusted effects on reading, although statistically 
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significant, were smaller (about +0.5% difference favoring NISL schools). In terms of 
unadjusted results, NISL schools gained 3.9% versus 0.7% for comparison sites in mathematics, 
and 4.1% versus 3.7% in reading/ELA. For the matched out-of-district elementary schools, 
significant trends favoring NISL over comparison schools were also indicated in both 
mathematics and reading/ELA. Significant quadratic effects in mathematics further revealed 
acceleration in the growth rate over time for NISL schools. 

Middle schools. As with the elementary school results, statistically significant positive 
effects of NISL status were observed for both mathematics and reading in the middle school 
sample. On an adjusted basis, NISL middle schools had about 2% more students scoring 
proficient or better in both mathematics and reading/ELA than comparison schools. As with 
elementary schools, a significant quadratic effect on mathematics school performance trends 
revealed that the rate of improvement was accelerating over time in NISL schools. 

High schools. In mathematics, NISL schools had statistically significantly higher rates of 
improvement, with adjusted differences in 2009 performance equal to 5%. In reading/ELA, 
covariate adjusted differences significantly favored NISL over comparison sites by about 1.5%, 
although overall adjusted performance declined between 2007 and 2009 for both groups of 
schools. 

Discussion 

NISL schools consistently surpassed the comparison schools in achievement gains at a 
statistically significant level from the baseline year of 2006 to 2009. A randomized experiment 
was not feasible given state and district policies for program implementation (e.g., see Appendix 
B). However, the present ex post facto design appears highly rigorous, particularly in 
minimizing validity threats frequently associated in evaluations of leadership programs with 
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sampling bias. Specifically, participants were described by the state and districts as being mixed 
in their experiences, success rates, and skills, with some targeted due to demonstrating strong 
potential for leadership and others due to needing professional development to address 
weaknesses. Also, the repeated-measures design treated nearly all principals as their own 
controls in analyzing school achievement patterns over time. 

Predictably, the achievement gains for NISL principals were strongest in 2008 and 2009 
as levels of participation in the NSL program (both number of principals and exposure) 
increased. The significant quadratic effects obtained in several of the analyses reflected this 
trend for program effects to accelerate over time. As depicted in Figure 17, across the four 
school cohorts examined (within-district-matched elementary schools, out-of-district matched 
elementary schools, middle schools, and high schools), NISL schools surpassed comparison 
schools in the percentage of students achieving proficiency in mathematics by 2.69%, 3.71%, 
1.70%, and 5.52%, respectively; and in reading/ELA by .37%, 2.55%, 1.63%, and 1.89%. Given 
that approximately 40,000 students were included in the combined samples, these advantages 
appear highly meaningful. For example, across the high school subsample alone, replication of 
the present NSL effects in mathematics and reading/ELA in similar schools would result in about 
275 and 103 more students achieving proficiency on the respective tests. Given that half of the 
principals began the program in 2007 and the other half not until 2008, the present usage of 2009 
as the most distant assessment year certainly seems likely to under-estimate potential program 
impacts. An additional consideration is that principals need time to implement new strategies in 
ways that impact teachers, who in turn, need time to improve instruction, learning, and 
achievement. Follow-up evaluation research of the present 2007 and 2008 cohorts, therefore, is 
strongly encouraged to determine post-program effects over a longer time period. 
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Appendix A: Supplementary Tables 



Table Al.A 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 
2006: Elementary Within-district Matches 















READING 


Pair 


Program 


FRL2006 


IEP2006 


LEP2006 


MATH 2006 


2006 


1 


Control 


.4444 


.0556 


.1111 


57.7000 


33.9333 




Treatment 


.4013 


.0382 


.0828 


51.4000 


33.7667 


2 


Control 


.4638 


.0298 


op 


63.3000 


40.8000 




Treatment 


.4267 


.0400 




66.7667 


47.3000 


3 


Control 


.4478 


.0597 




71.8333 


61.0333 




Treatment 


.3654 


.0481 




60.4667 


41.0000 


4 


Control 


.1441 


.0721 




82.6667 


66.9000 




Treatment 


.1429 


.0771 




83.0000 


65.5333 


5 


Control 


.1515 




mm 


70.9000 


61.4333 




Treatment 


.1192 






74.0333 


68.2333 


6 


Control 


.0479 


.0691 




88.7000 


80.5000 




Treatment 


.0870 


.0543 




94.2667 


85.9667 


7 


Control 


.1322 




.0000 


75.4333 


66.1667 




Treatment 


.1263 


.1579 


.0000 


74.2667 


67.6333 


8 


Control 


.0637 






87.5667 


79.1000 




Treatment 


.0625 






83.7667 


78.8333 


9 


Control 


.0616 


.1233 


.0000 


90.3000 


82.1000 




Treatment 


.0813 




.0041 


89.9333 


84.6000 


10 


Control 


.0333 


.0333 


.0000 


90.6333 


82.1000 




Treatment 


.0652 


.0000 


.0000 


91.7333 


78.4333 


11 


Control 


.0588 


.0515 


.0000 


95.6750 


85.4750 




Treatment 


.0665 


.0190 


.0000 


88.4250 


81.5500 


12 


Control 


.3165 


.0935 




83.8000 


70.1333 




Treatment 


.2222 


.0824 




73.7333 


61.4333 


13 


Control 


.1022 


.0311 




88.9000 


83.9750 




Treatment 


.0405 






91.2000 


86.0250 
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Table Al.A Continued 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 



2006: Elementary Within-district Matches 



14 


Control 


.1222 


.0333 


.0000 


93.5500 


82.1500 




Treatment 


.0588 


.0441 


.0000 


84.7750 


79.2500 


15 


Control 


.0909 






84.7500 


81.2500 




Treatment 


.0601 






86.0667 


83.6333 


16 


Control 


.3711 






51.9000 


37.2667 




Treatment 


.3711 




.1753 


52.0333 


32.1667 


17 


Control 


.0661 




.0331 


83.8333 


78.3000 




Treatment 


.1209 




.0549 


83.6667 


73.1000 


18 


Control 


.0240 


.0240 


.0080 


91.2000 


79.4333 




Treatment 


.0342 


.0160 


.0023 


87.2667 


79.4667 


19 


Control 


.2151 






82.9333 


68.4333 




Treatment 


.2273 


.0341 




83.0000 


72.1333 


20 


Control 


.0882 




.0000 


90.2000 


83.9000 




Treatment 


.0090 




.0000 


88.8000 


86.1500 


21 


Control 


.0067 






93.6667 


89.0333 




Treatment 


.0067 






92.5333 


90.7333 


22 


Control 


.1078 




.0000 


92.1500 


82.1500 




Treatment 


.1048 




.0000 


91.3000 


85.4000 


23 


Control 


.1338 




.0000 


83.2667 


73.5333 




Treatment 


.1901 


.1157 


.0083 


88.3000 


73.2667 


24 


Control 


.3538 


.0769 


.0000 


36.5750 


25.2000 




Treatment 


.3245 


.0479 


.0000 


31.7750 


29.0500 


25 


Control 


.2867 


.0559 




56.3500 


44.8000 




Treatment 


.3099 


.0643 




59.2250 


41.0750 


26 


Control 


.2326 






74.4333 


62.0333 




Treatment 


.2617 






76.8000 


68.4500 


27 


Control 


.0531 


.0435 


.0000 


88.4000 


81.3500 




Treatment 


.0313 


.0208 


.0000 


92.8250 


81.9500 


28 


Control 


.2500 




.0000 


86.9000 


70.9333 




Treatment 


.0275 




.0138 


59.6000 


53.6000 


29 


Control 


.2222 


.0833 


.0000 


94.6000 


80.1000 




Treatment 


.1216 


.0811 


.0000 


89.0333 


73.7667 
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Table Al.A Continued 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 



2006: Elementary Within-district Matches 



30 


Control 


.1166 


.0429 




79.9667 


73.8000 




Treatment 


.1495 


.0935 




79.4333 


74.6333 


31 


Control 


.2111 


.0889 


.0000 


84.0000 


67.6667 




Treatment 


.1333 


.0571 


.0000 


73.1333 


61.0000 


32 


Control 


.2222 


.1111 


.0000 


88.1000 


77.3000 




Treatment 


.1714 


.1286 


.0000 


90.9000 


80.8500 


33 


Control 


.1259 


.0769 


.0140 


88.8750 


79.4500 




Treatment 


.1127 


.0211 


.0563 


80.6750 


73.5250 


34 


Control 


.1410 




.0000 


97.8750 


77.7000 




Treatment 


.0750 




.0000 


87.8250 


79.9000 


35 


Control 


.2133 




.0000 


75.6000 


65.5000 




Treatment 


.2813 




.0000 


77.7000 


65.9667 


36 


Control 


.2901 


.1221 


.0000 


51.2750 


36.4000 




Treatment 


.3059 


.1176 


.0000 


50.0000 


32.4500 


Total 


Control 


.1781 




mm 


80.4947 


69.2037 




Treatment 


.1582 




m 


78.0461 


68.1063 
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Table Al.B 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 2006: 
Elementary Out-of-district Matches 



READING 



Pair 


Program 


FRL2006 


IEP2006 


LEP2006 


MATH 2006 


2006 


101 


Control 


.3545 


.0818 


.0000 


71.1000 


67.1250 




Treatment 


.4815 


.1852 


.0000 


86.0500 


78.4500 


102 


Control 


.2161 




.0073 


58.9000 


44.9250 




Treatment 


.2066 




.0000 


57.8500 


47.4750 


103 


Control 


.3981 


.1748 


.0000 


81.8500 


58.1000 




Treatment 


.3810 




.0000 


79.8333 


64.2333 


104 


Control 


.1949 




.0000 


61.7500 


50.6250 




Treatment 


.2692 


.1154 


.0000 


70.3667 


60.5667 


105 


Control 


.3333 






87.1000 


72.8000 




Treatment 


.1429 






64.4000 


64.8250 


106 


Control 


.2755 




.1472 


85.0000 


62.8000 




Treatment 


.2692 


MM 




84.1667 


74.0000 


107 


Control 


.1856 


.0838 


.0000 


76.2250 


63.2750 




Treatment 


.1548 


.0595 


.0000 


72.5000 


62.1000 


108 


Control 


.0258 


.0774 


.0000 


57.8333 


50.4667 




Treatment 


.1313 


.0707 


.0000 


70.2750 


70.2250 


109 


Control 


.1935 




.0000 


78.5000 


62.0000 




Treatment 


.1975 


.1176 


.0000 


79.0000 


69.8667 


110 


Control 


.1388 




.0000 


72.9500 


68.9750 




Treatment 


.2195 




.0000 


82.5000 


73.6667 


111 


Control 


.2614 


.0980 




88.0500 


69.9500 




Treatment 


.1293 


.0476 




72.4000 


73.2500 


112 


Control 


.1719 


.0078 


.0000 


78.7250 


70.5000 




Treatment 


.1022 


.0584 


.0000 


70.4500 


50.0250 


113 


Control 


.1181 


.0945 




76.8667 


66.4000 




Treatment 


.1533 


.0657 




81.0333 


74.4000 


114 


Control 


.1610 


.0341 


.0000 


82.4000 


79.4000 




Treatment 


.1698 


.1792 


.0000 


83.4500 


72.4000 
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Table Al.B Continued 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 
2006: Elementary Out-of-district Matches 



115 


Control 


.2895 


.2105 


.0000 


98.7000 


85.6000 




Treatment 


.0968 


.0323 


.0000 


75.8750 


75.2250 




Total 


.1931 


.1214 


.0000 


87.2875 


80.4125 


116 


Control 


.1557 






83.1000 


70.3333 




Treatment 


.2138 






90.0000 


78.7000 


117 


Control 


.0952 






76.2000 


65.6750 




Treatment 


.1105 






78.0250 


65.7750 


118 


Control 


.2025 


.0331 


.0248 


90.2250 


76.7000 




Treatment 


.1654 


.0451 




85.8500 


71.1500 


119 


Control 


.1618 


.1324 




86.4333 


79.7000 




Treatment 


.0739 






76.0333 


68.6333 


120 


Control 


.1083 


.0583 


.0000 


80.5333 


65.9667 




Treatment 


.0897 


.0552 


.0000 


78.3333 


62.9667 




Total 


.0990 


.0568 


.0000 


79.4333 


64.4667 


121 


Control 


.2376 


.1287 


.0000 


96.0333 


88.1333 




Treatment 


.1136 




.0000 


81.4250 


65.1250 


122 


Control 


.0708 






77.6667 


80.6000 




Treatment 


.1204 






83.5500 


73.5500 


123 


Control 


.0645 




.0000 


78.7500 


67.1000 




Treatment 


.1000 




.0000 


82.9333 


67.5333 


124 


Control 


.1316 


mm 


.0000 


88.2500 


79.1500 




Treatment 


.1529 


■n 


.0000 


90.8000 


82.7000 


125 


Control 


.0968 






85.3000 


76.4000 




Treatment 


.0299 






77.3500 


66.6000 


126 


Control 


.0559 


.0503 


.0000 


83.9000 


71.5250 




Treatment 


.1379 


.1149 


.0000 


93.5667 


74.3333 


127 


Control 


.0397 


.0464 


.0000 


85.6000 


72.3000 




Treatment 


.0267 


.0600 


.0000 


84.1000 


63.1500 
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Table Al.B Continued 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 
2006: Elementary Out-of-district Matches 



128 


Control 


.0386 


.0858 


.0043 


92.5500 


86.6750 




Treatment 


.0500 


.0450 


.0000 


93.8667 


83.1000 


129 


Control 


.0328 


.0000 


.0000 


94.0333 


86.8667 




Treatment 


.0138 


.0690 


.0000 


91.8000 


83.8667 


130 


Control 


.0023 


.0254 


.0000 


95.4500 


92.4500 




Treatment 


.0068 


.0473 


.0000 


96.0000 


87.1333 


131 


Control 


.3256 


.1349 




15.2000 


10.0500 




Treatment 


.4935 






35.9667 


27.1000 


132 


Control 


.3074 


.0736 




63.9667 


56.3333 




Treatment 


.4167 


.0389 




77.0333 


59.9333 


Total 


Control 


.1702 


.0762 




79.0357 


68.7156 




Treatment 


.1694 


.0747 




78.9620 


68.5018 
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Table A2 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 2006: Middle 



Schools 



Pair 


Program 


FRL2006 


IEP2006 


LEP2006 


Math 2006 Reading 2006 


1.00 


Control 


.6856 


.1005 


.0515 


44.3667 


38.4000 




Treatment 


.8333 


.1667 


.0000 


43.7667 


42.1667 


2.00 


Control 


.5279 


.1416 


.0858 


54.4333 


54.3000 




Treatment 


.4961 


.2126 


.0157 


50.2333 


56.0000 


3.00 


Control 


.5399 


.1411 


.0000 


58.2333 


57.6000 




Treatment 


.4463 


.1653 


.0000 


59.8500 


50.1000 


4.00 


Control 


.4301 


.0968 


.0000 


69.0500 


62.8500 




Treatment 


.3455 


.2042 


.0000 


66.3000 


59.8000 


5.00 


Control 


.4118 


.1216 


.0039 


66.2500 


72.0500 




Treatment 


.2841 


.1023 


.0000 


58.2667 


72.1333 


6.00 


Control 


.2159 


.1023 


.0057 


62.3500 


69.3000 




Treatment 


.3200 


.1100 


.0000 


65.4333 


72.9333 


7.00 


Control 


.4110 


.1166 


.0000 


73.9333 


71.3333 




Treatment 


.3243 


.0270 


.0000 


68.1000 


71.4500 


8.00 


Control 


.2870 


.1204 


.0000 


66.4000 


72.5500 




Treatment 


.1667 


.1071 


.0000 


65.1333 


66.3667 


9.00 


Control 


.2346 


.2308 


.0115 


66.3333 


70.5000 




Treatment 


.2171 


.1318 


.0000 


64.6500 


71.4000 


10.00 


Control 


.1713 


.1528 


.0139 


69.4000 


72.5333 




Treatment 


.2452 


.1497 


.0223 


73.5333 


73.3333 


11.00 


Control 


.2041 


.0867 


.0051 


73.6667 


72.6333 




Treatment 


.1587 


.1346 


.0000 


72.5667 


71.0333 


12.00 


Control 


.1878 


.1224 


.0020 


73.8000 


75.4000 




Treatment 


.1027 


.1712 


.0205 


73.2000 


70.6500 


13.00 


Control 


.2879 


.1364 


.0000 


80.3500 


76.5500 




Treatment 


.2421 


.1230 


.0198 


78.2667 


75.8667 


14.00 


Control 


.1423 


.0751 


.0040 


70.2333 


81.2333 




Treatment 


.1815 


.1231 


.0000 


79.9500 


74.3000 
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Table A2 

Selected 2006 Characteristics of Matched Comparison and NISL Schools 
in 2006: Middle Schools 



15.00 


Control 


.0000 


.1224 


.0000 


73.8000 


71.5500 




Treatment 


.1480 


.1704 


.0224 


74.9000 


79.8000 


16.00 


Control 


.1225 




.0000 


75.4000 


82.3500 




Treatment 


.1721 


.1148 


.0000 


76.6500 


84.2000 


17.00 


Control 


.1824 




.0063 


80.4333 


83.5333 




Treatment 


.2316 


.1263 


.0000 


83.5333 


83.6667 


18.00 


Control 


.0943 


.1639 


.0123 


90.0500 


88.9500 




Treatment 


.0367 




.0000 


86.7000 


88.9000 


19.00 


Control 


.3474 




.0000 


55.7667 


66.7000 




Treatment 


.2941 


.1471 


.0000 


55.3333 


63.5333 


Total 


Control 


.2458 


.1210 


.0053 


71.9996 


73.1694 




Treatment 


.2761 


.1345 


.0053 


68.2298 


69.8754 
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Table A3 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 
2006: High Schools 



Pair 


Program 


FRL2006 


IEP2006 


LEP2006 


Math 2006 


Reading 2006 


1.00 


Control 


.4309 


.0994 


.0331 


16.2000 


28.1000 




Treatment 


.4692 


.2000 


.0000 


15.2000 


32.4000 




Total 


.4597 


.1750 


.0083 


15.4491 


31.3290 


2.00 


Control 


1.0000 


.0469 


.0000 


37.6000 


51.6000 




Treatment 


.5217 


.1478 


.0000 


21.6000 


34.3000 




Total 


.7488 


.0999 


.0000 


29.1948 


42.5119 


3.00 


Control 


.4007 




.0000 


44.8000 


59.0000 




Treatment 


.2021 




.0000 


33.7000 


56.5000 




Total 


.3068 




.0000 


39.5514 


57.8179 


4.00 


Control 


.2667 


.1778 


.0000 


40.0000 


57.7000 




Treatment 


.3562 


.1003 


.0000 


43.8000 


60.4000 




Total 


.3065 


.1433 


.0000 


41.6910 


58.9015 


5.00 


Control 


.3082 


.0688 


.0291 


47.2000 


61.3000 




Treatment 


.2418 


.2198 


.0000 


47.2000 


56.2000 




Total 


.2589 


.1808 


.0075 


47.2000 


57.5162 


6.00 


Control 


.2804 


.0847 


.0000 


46.7000 


63.4000 




Treatment 


.1348 


.1028 


.0000 


39.4000 


60.6000 




Total 


.2106 


.0934 


.0000 


43.1985 


62.0570 


7.00 


Control 


.4198 


.1728 




61.0000 


61.9000 




Treatment 


.1731 


.1635 




47.9000 


57.9000 




Total 


.3006 


.1683 




54.6722 


59.9679 


8.00 


Control 


.3196 


.1237 


.0000 


45.2000 


74.2000 




Treatment 


.2254 


.1619 


.0000 


49.2000 


62.9000 




Total 


.2762 


.1413 


.0000 


47.0439 


68.9910 


9.00 


Control 


.0018 


.0458 


.0000 


33.7000 


66.7000 




Treatment 


.1389 


.1056 


.0000 


48.5000 


60.8000 




Total 


.0641 


.0730 


.0000 


40.4276 


64.0180 
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Table A3 Continued 

Selected 2006 Characteristics of Matched Comparison and NISL Schools in 
2006: High Schools 



10.00 


Control 


.2690 


.1066 


.0000 


56.8000 


66.0000 




Treatment 


.2917 


.1528 


.0000 


52.1000 


73.2000 




Total 


.2800 


.1290 


.0000 


54.5175 


69.4966 


11.00 


Control 


.0725 


.0966 




49.2000 


61.1000 




Treatment 


.0850 


.1275 


.0033 


49.7000 


61.3000 




Total 


.0783 


.1111 


.0015 


49.4348 


61.1939 


12.00 


Control 


.0601 


.0687 




48.4000 


68.4000 




Treatment 


.1323 


.1245 


.0078 


54.4000 


67.2000 




Total 


.0941 


.0950 


.0037 


51.2286 


67.8343 


13.00 


Control 


.2588 


.1529 


.0118 


65.1000 


74.6000 




Treatment 


.0854 


.1357 


.0000 


56.3000 


71.6000 




Total 


.1568 


.1428 


.0048 


59.9218 


72.8347 


14.00 


Control 


.0968 


.1226 


.0000 


62.8000 


73.9000 




Treatment 


.2857 


.0794 


.0000 


70.9000 


79.0000 




Total 


.1865 


.1021 


.0000 


66.6472 


76.3223 


Total 


Control 


.2786 


.1092 


.0022 


47.8340 


63.7732 




Treatment 


.2388 


.1430 


.0022 


44.9929 


59.5929 




Total 


.2584 


.1264 


.0022 


46.3920 


61.6515 
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Appendix B: Evaluation Rubric for Selecting Program Participants 

PA INSPIRED LEADERSHIP INITIATIVE 

Name_ 

Position. 

District, 

Intermediate Unit 



Participant Selection Rubric 


Score 


Instructional Leadership Responsibilities (0 points) 


required 


Page One School (1 point) 




District/School is in School Improvement or Corrective Action (5 
points) 




District is in Phase II of Strategic Planning (1 point) 




Principal (2 points for first year, 3 points for second year, 
4 points for 3 rd year) 




Assistant Principal (1 point) 




Signed support statement from direct supervisor (0 points) 


required 


First principal to participate from district. (1 point) 




Total 







